25 research outputs found

    Designing a conversational requirements elicitation system for end-users

    Get PDF
    [Context] Digital transformation impacts an ever-increasing degree of everyone’s business and private life. It is imperative to incorporate a wide audience of user requirements in the development process to design successful information systems (IS). Hence, requirements elicitation (RE) is increasingly performed by end-users that are novices at contributing requirements to IS development projects. [Objective] We need to develop RE systems that are capable of assisting a wide audience of end-users in communicating their needs and requirements. Prominent methods, such as elicitation interviews, are challenging to apply in such a context, as time and location constraints limit potential audiences. [Research Method] The presented dissertation project utilizes design science research to develop a requirements self-elicitation system, LadderBot. A conversational agent (CA) enables end-users to articulate needs and requirements on the grounds of the laddering method. The CA mimics a human interviewer’s capability to rephrase questions and provide assistance in the process and allows users to converse in their natural language. Furthermore, the tool will assist requirements analysts with the subsequent aggregation and analysis of collected data. [Contribution] The dissertation project makes a practical contribution in the form of a ready-to-use system for wide audience end-user RE and subsequent analysis utilizing laddering as cognitive elicitation technique. A theoretical contribution is provided by developing a design theory for the application of conversational agents for RE, including the laboratory and field evaluation of design principles

    Designing AI-Based Systems for Qualitative Data Collection and Analysis

    Get PDF
    With the continuously increasing impact of information systems (IS) on private and professional life, it has become crucial to integrate users in the IS development process. One of the critical reasons for failed IS projects is the inability to accurately meet user requirements, resulting from an incomplete or inaccurate collection of requirements during the requirements elicitation (RE) phase. While interviews are the most effective RE technique, they face several challenges that make them a questionable fit for the numerous, heterogeneous, and geographically distributed users of contemporary IS. Three significant challenges limit the involvement of a large number of users in IS development processes today. Firstly, there is a lack of tool support to conduct interviews with a wide audience. While initial studies show promising results in utilizing text-based conversational agents (chatbots) as interviewer substitutes, we lack design knowledge for designing AI-based chatbots that leverage established interviewing techniques in the context of RE. By successfully applying chatbot-based interviewing, vast amounts of qualitative data can be collected. Secondly, there is a need to provide tool support enabling the analysis of large amounts of qualitative interview data. Once again, while modern technologies, such as machine learning (ML), promise remedy, concrete implementations of automated analysis for unstructured qualitative data lag behind the promise. There is a need to design interactive ML (IML) systems for supporting the coding process of qualitative data, which centers around simple interaction formats to teach the ML system, and transparent and understandable suggestions to support data analysis. Thirdly, while organizations rely on online feedback to inform requirements without explicitly conducting RE interviews (e.g., from app stores), we know little about the demographics of who is giving feedback and what motivates them to do so. Using online feedback as requirement source risks including solely the concerns and desires of vocal user groups. With this thesis, I tackle these three challenges in two parts. In part I, I address the first and the second challenge by presenting and evaluating two innovative AI-based systems, a chatbot for requirements elicitation and an IML system to semi-automate qualitative coding. In part II, I address the third challenge by presenting results from a large-scale study on IS feedback engagement. With both parts, I contribute with prescriptive knowledge for designing AI-based qualitative data collection and analysis systems and help to establish a deeper understanding of the coverage of existing data collected from online sources. Besides providing concrete artifacts, architectures, and evaluations, I demonstrate the application of a chatbot interviewer to understand user values in smartphones and provide guidance for extending feedback coverage from underrepresented IS user groups

    LadderBot: A requirements self-elicitation system

    Get PDF
    Digital transformation impacts an ever-increasing amount of everyone’s business and private life. It is imperative to incorporate user requirements in the development process to design successful information systems (IS). Hence, requirements elicitation (RE) is increasingly performed by users that are novices at contributing requirements to IS development projects. [Objective] We need to develop RE systems that are capable of assisting a wide audience of users in communicating their needs and requirements. Prominent methods, such as elicitation interviews, are challenging to apply in such a context, as time and location constraints limit potential audiences. [Research Method] We present the prototypical self-elicitation system “LadderBot”. A conversational agent (CA) enables end-users to articulate needs and requirements on the grounds of the laddering method. The CA mimics a human (expert) interviewer’s capability to rephrase questions and provide assistance in the process. An experimental study is proposed to evaluate LadderBot against an established questionnaire-based laddering approach. [Contribution] This work-in-progress introduces the chatbot LadderBot as a tool to guide novice users during requirements self-elicitation using the laddering technique. Furthermore, we present the design of an experimental study and outline the next steps and a vision for the future

    Cody: An AI-Based System to Semi-Automate Coding for Qualitative Research

    Get PDF
    Qualitative research can produce a rich understanding of a phenomenon but requires an essential and strenuous data annotation process known as coding. Coding can be repetitive and time-consuming, particularly for large datasets. Existing AI-based approaches for partially automating coding, like supervised machine learning (ML) or explicit knowledge represented in code rules, require high technical literacy and lack transparency. Further, little is known about the interaction of researchers with AI-based coding assistance. We introduce Cody, an AI-based system that semi-automates coding through code rules and supervised ML. Cody supports researchers with interactively (re)defining code rules and uses ML to extend coding to unseen data. In two studies with qualitative researchers, we found that (1) code rules provide structure and transparency, (2) explanations are commonly desired but rarely used, (3) suggestions benefit coding quality rather than coding speed, increasing the intercoder reliability, calculated with Krippendorff’s Alpha, from 0.085 (MAXQDA) to 0.33 (Cody)

    Cody: An Interactive Machine Learning System for Qualitative Coding

    Get PDF
    Qualitative coding, the process of assigning labels to text as part of qualitative analysis, is time-consuming and repetitive, especially for large datasets. While available QDAS sometimes allows the semi-automated extension of annotations to unseen data, recent user studies revealed critical issues. In particular, the integration of automated code suggestions into the coding process is not transparent and interactive. In this work, we present Cody, a system for semi-automated qualitative coding that suggests codes based on human-defined coding rules and supervised machine learning (ML). Suggestions and rules can be revised iteratively by users in a lean interface that provides explanations for code suggestions. In a preliminary evaluation, 42% of all documents could be coded automatically based on code rules. Cody is the first coding system to allow users to define query-style code rules in combination with supervised ML. Thereby, users can extend manual annotations to unseen data to improve coding speed and quality

    The Impact of Anthropomorphic and Functional Chatbot Design Features in Enterprise Collaboration Systems on User Acceptance

    Get PDF
    Information technology is rapidly changing the way how people collaborate in enterprises. Chatbots integrated into enterprise collaboration systems can strengthen collaboration culture and help reduce work overload. In light of a growing usage of chatbots in enterprise collaboration systems, we examine the influence of anthropomorphic and functional chatbot design features on user acceptance. We conducted a survey with professionals familiar with interacting with chatbots in a work environment. The results show a significant effect of anthropomorphic design features on perceived usefulness, with a strength four times the size of the effect of functional chatbot features. We suggest that researchers and practitioners alike dedicate priorities to anthropomorphic design features with the same magnitude as common for functional design features in chatbot design and research

    Human Practice. Digital Ecologies. Our Future. : 14. Internationale Tagung Wirtschaftsinformatik (WI 2019) : Tagungsband

    Get PDF
    Erschienen bei: universi - UniversitĂ€tsverlag Siegen. - ISBN: 978-3-96182-063-4Aus dem Inhalt: Track 1: Produktion & Cyber-Physische Systeme Requirements and a Meta Model for Exchanging Additive Manufacturing Capacities Service Systems, Smart Service Systems and Cyber- Physical Systems—What’s the difference? Towards a Unified Terminology Developing an Industrial IoT Platform – Trade-off between Horizontal and Vertical Approaches Machine Learning und Complex Event Processing: Effiziente Echtzeitauswertung am Beispiel Smart Factory Sensor retrofit for a coffee machine as condition monitoring and predictive maintenance use case Stakeholder-Analyse zum Einsatz IIoT-basierter Frischeinformationen in der Lebensmittelindustrie Towards a Framework for Predictive Maintenance Strategies in Mechanical Engineering - A Method-Oriented Literature Analysis Development of a matching platform for the requirement-oriented selection of cyber physical systems for SMEs Track 2: Logistic Analytics An Empirical Study of Customers’ Behavioral Intention to Use Ridepooling Services – An Extension of the Technology Acceptance Model Modeling Delay Propagation and Transmission in Railway Networks What is the impact of company specific adjustments on the acceptance and diffusion of logistic standards? Robust Route Planning in Intermodal Urban Traffic Track 3: Unternehmensmodellierung & Informationssystemgestaltung (Enterprise Modelling & Information Systems Design) Work System Modeling Method with Different Levels of Specificity and Rigor for Different Stakeholder Purposes Resolving Inconsistencies in Declarative Process Models based on Culpability Measurement Strategic Analysis in the Realm of Enterprise Modeling – On the Example of Blockchain-Based Initiatives for the Electricity Sector Zwischenbetriebliche Integration in der Möbelbranche: Konfigurationen und Einflussfaktoren Novices’ Quality Perceptions and the Acceptance of Process Modeling Grammars Entwicklung einer Definition fĂŒr Social Business Objects (SBO) zur Modellierung von Unternehmensinformationen Designing a Reference Model for Digital Product Configurators Terminology for Evolving Design Artifacts Business Role-Object Specification: A Language for Behavior-aware Structural Modeling of Business Objects Generating Smart Glasses-based Information Systems with BPMN4SGA: A BPMN Extension for Smart Glasses Applications Using Blockchain in Peer-to-Peer Carsharing to Build Trust in the Sharing Economy Testing in Big Data: An Architecture Pattern for a Development Environment for Innovative, Integrated and Robust Applications Track 4: Lern- und Wissensmanagement (e-Learning and Knowledge Management) eGovernment Competences revisited – A Literature Review on necessary Competences in a Digitalized Public Sector Say Hello to Your New Automated Tutor – A Structured Literature Review on Pedagogical Conversational Agents Teaching the Digital Transformation of Business Processes: Design of a Simulation Game for Information Systems Education Conceptualizing Immersion for Individual Learning in Virtual Reality Designing a Flipped Classroom Course – a Process Model The Influence of Risk-Taking on Knowledge Exchange and Combination Gamified Feedback durch Avatare im Mobile Learning Alexa, Can You Help Me Solve That Problem? - Understanding the Value of Smart Personal Assistants as Tutors for Complex Problem Tasks Track 5: Data Science & Business Analytics Matching with Bundle Preferences: Tradeoff between Fairness and Truthfulness Applied image recognition: guidelines for using deep learning models in practice Yield Prognosis for the Agrarian Management of Vineyards using Deep Learning for Object Counting Reading Between the Lines of Qualitative Data – How to Detect Hidden Structure Based on Codes Online Auctions with Dual-Threshold Algorithms: An Experimental Study and Practical Evaluation Design Features of Non-Financial Reward Programs for Online Reviews: Evaluation based on Google Maps Data Topic Embeddings – A New Approach to Classify Very Short Documents Based on Predefined Topics Leveraging Unstructured Image Data for Product Quality Improvement Decision Support for Real Estate Investors: Improving Real Estate Valuation with 3D City Models and Points of Interest Knowledge Discovery from CVs: A Topic Modeling Procedure Online Product Descriptions – Boost for your Sales? EntscheidungsunterstĂŒtzung durch historienbasierte Dienstreihenfolgeplanung mit Pattern A Semi-Automated Approach for Generating Online Review Templates Machine Learning goes Measure Management: Leveraging Anomaly Detection and Parts Search to Improve Product-Cost Optimization Bedeutung von Predictive Analytics fĂŒr den theoretischen Erkenntnisgewinn in der IS-Forschung Track 6: Digitale Transformation und Dienstleistungen Heuristic Theorizing in Software Development: Deriving Design Principles for Smart Glasses-based Systems Mirroring E-service for Brick and Mortar Retail: An Assessment and Survey Taxonomy of Digital Platforms: A Platform Architecture Perspective Value of Star Players in the Digital Age Local Shopping Platforms – Harnessing Locational Advantages for the Digital Transformation of Local Retail Outlets: A Content Analysis A Socio-Technical Approach to Manage Analytics-as-a-Service – Results of an Action Design Research Project Characterizing Approaches to Digital Transformation: Development of a Taxonomy of Digital Units Expectations vs. Reality – Benefits of Smart Services in the Field of Tension between Industry and Science Innovation Networks and Digital Innovation: How Organizations Use Innovation Networks in a Digitized Environment Characterising Social Reading Platforms— A Taxonomy-Based Approach to Structure the Field Less Complex than Expected – What Really Drives IT Consulting Value Modularity Canvas – A Framework for Visualizing Potentials of Service Modularity Towards a Conceptualization of Capabilities for Innovating Business Models in the Industrial Internet of Things A Taxonomy of Barriers to Digital Transformation Ambidexterity in Service Innovation Research: A Systematic Literature Review Design and success factors of an online solution for cross-pillar pension information Track 7: IT-Management und -Strategie A Frugal Support Structure for New Software Implementations in SMEs How to Structure a Company-wide Adoption of Big Data Analytics The Changing Roles of Innovation Actors and Organizational Antecedents in the Digital Age Bewertung des Kundennutzens von Chatbots fĂŒr den Einsatz im Servicedesk Understanding the Benefits of Agile Software Development in Regulated Environments Are Employees Following the Rules? On the Effectiveness of IT Consumerization Policies Agile and Attached: The Impact of Agile Practices on Agile Team Members’ Affective Organisational Commitment The Complexity Trap – Limits of IT Flexibility for Supporting Organizational Agility in Decentralized Organizations Platform Openness: A Systematic Literature Review and Avenues for Future Research Competence, Fashion and the Case of Blockchain The Digital Platform Otto.de: A Case Study of Growth, Complexity, and Generativity Track 8: eHealth & alternde Gesellschaft Security and Privacy of Personal Health Records in Cloud Computing Environments – An Experimental Exploration of the Impact of Storage Solutions and Data Breaches Patientenintegration durch Pfadsysteme Digitalisierung in der StressprĂ€vention – eine qualitative Interviewstudie zu Nutzenpotenzialen User Dynamics in Mental Health Forums – A Sentiment Analysis Perspective Intent and the Use of Wearables in the Workplace – A Model Development Understanding Patient Pathways in the Context of Integrated Health Care Services - Implications from a Scoping Review Understanding the Habitual Use of Wearable Activity Trackers On the Fit in Fitness Apps: Studying the Interaction of Motivational Affordances and Users’ Goal Orientations in Affecting the Benefits Gained Gamification in Health Behavior Change Support Systems - A Synthesis of Unintended Side Effects Investigating the Influence of Information Incongruity on Trust-Relations within Trilateral Healthcare Settings Track 9: Krisen- und KontinuitĂ€tsmanagement Potentiale von IKT beim Ausfall kritischer Infrastrukturen: Erwartungen, Informationsgewinnung und Mediennutzung der Zivilbevölkerung in Deutschland Fake News Perception in Germany: A Representative Study of People’s Attitudes and Approaches to Counteract Disinformation Analyzing the Potential of Graphical Building Information for Fire Emergency Responses: Findings from a Controlled Experiment Track 10: Human-Computer Interaction Towards a Taxonomy of Platforms for Conversational Agent Design Measuring Service Encounter Satisfaction with Customer Service Chatbots using Sentiment Analysis Self-Tracking and Gamification: Analyzing the Interplay of Motivations, Usage and Motivation Fulfillment Erfolgsfaktoren von Augmented-Reality-Applikationen: Analyse von Nutzerrezensionen mit dem Review-Mining-Verfahren Designing Dynamic Decision Support for Electronic Requirements Negotiations Who is Stressed by Using ICTs? A Qualitative Comparison Analysis with the Big Five Personality Traits to Understand Technostress Walking the Middle Path: How Medium Trade-Off Exposure Leads to Higher Consumer Satisfaction in Recommender Agents Theory-Based Affordances of Utilitarian, Hedonic and Dual-Purposed Technologies: A Literature Review Eliciting Customer Preferences for Shopping Companion Apps: A Service Quality Approach The Role of Early User Participation in Discovering Software – A Case Study from the Context of Smart Glasses The Fluidity of the Self-Concept as a Framework to Explain the Motivation to Play Video Games Heart over Heels? An Empirical Analysis of the Relationship between Emotions and Review Helpfulness for Experience and Credence Goods Track 11: Information Security and Information Privacy Unfolding Concerns about Augmented Reality Technologies: A Qualitative Analysis of User Perceptions To (Psychologically) Own Data is to Protect Data: How Psychological Ownership Determines Protective Behavior in a Work and Private Context Understanding Data Protection Regulations from a Data Management Perspective: A Capability-Based Approach to EU-GDPR On the Difficulties of Incentivizing Online Privacy through Transparency: A Qualitative Survey of the German Health Insurance Market What is Your Selfie Worth? A Field Study on Individuals’ Valuation of Personal Data Justification of Mass Surveillance: A Quantitative Study An Exploratory Study of Risk Perception for Data Disclosure to a Network of Firms Track 12: Umweltinformatik und nachhaltiges Wirtschaften KommunikationsfĂ€den im Nadelöhr – Fachliche Prozessmodellierung der Nachhaltigkeitskommunikation am Kapitalmarkt Potentiale und Herausforderungen der Materialflusskostenrechnung Computing Incentives for User-Based Relocation in Carsharing Sustainability’s Coming Home: Preliminary Design Principles for the Sustainable Smart District Substitution of hazardous chemical substances using Deep Learning and t-SNE A Hierarchy of DSMLs in Support of Product Life-Cycle Assessment A Survey of Smart Energy Services for Private Households Door-to-Door Mobility Integrators as Keystone Organizations of Smart Ecosystems: Resources and Value Co-Creation – A Literature Review Ein EntscheidungsunterstĂŒtzungssystem zur ökonomischen Bewertung von Mieterstrom auf Basis der Clusteranalyse Discovering Blockchain for Sustainable Product-Service Systems to enhance the Circular Economy Digitale RĂŒckverfolgbarkeit von Lebensmitteln: Eine verbraucherinformatische Studie Umweltbewusstsein durch audiovisuelles Content Marketing? Eine experimentelle Untersuchung zur Konsumentenbewertung nachhaltiger Smartphones Towards Predictive Energy Management in Information Systems: A Research Proposal A Web Browser-Based Application for Processing and Analyzing Material Flow Models using the MFCA Methodology Track 13: Digital Work - Social, mobile, smart On Conversational Agents in Information Systems Research: Analyzing the Past to Guide Future Work The Potential of Augmented Reality for Improving Occupational First Aid Prevent a Vicious Circle! The Role of Organizational IT-Capability in Attracting IT-affine Applicants Good, Bad, or Both? Conceptualization and Measurement of Ambivalent User Attitudes Towards AI A Case Study on Cross-Hierarchical Communication in Digital Work Environments ‘Show Me Your People Skills’ - Employing CEO Branding for Corporate Reputation Management in Social Media A Multiorganisational Study of the Drivers and Barriers of Enterprise Collaboration Systems-Enabled Change The More the Merrier? The Effect of Size of Core Team Subgroups on Success of Open Source Projects The Impact of Anthropomorphic and Functional Chatbot Design Features in Enterprise Collaboration Systems on User Acceptance Digital Feedback for Digital Work? Affordances and Constraints of a Feedback App at InsurCorp The Effect of Marker-less Augmented Reality on Task and Learning Performance Antecedents for Cyberloafing – A Literature Review Internal Crowd Work as a Source of Empowerment - An Empirical Analysis of the Perception of Employees in a Crowdtesting Project Track 14: GeschĂ€ftsmodelle und digitales Unternehmertum Dividing the ICO Jungle: Extracting and Evaluating Design Archetypes Capturing Value from Data: Exploring Factors Influencing Revenue Model Design for Data-Driven Services Understanding the Role of Data for Innovating Business Models: A System Dynamics Perspective Business Model Innovation and Stakeholder: Exploring Mechanisms and Outcomes of Value Creation and Destruction Business Models for Internet of Things Platforms: Empirical Development of a Taxonomy and Archetypes Revitalizing established Industrial Companies: State of the Art and Success Principles of Digital Corporate Incubators When 1+1 is Greater than 2: Concurrence of Additional Digital and Established Business Models within Companies Special Track 1: Student Track Investigating Personalized Price Discrimination of Textile-, Electronics- and General Stores in German Online Retail From Facets to a Universal Definition – An Analysis of IoT Usage in Retail Is the Technostress Creators Inventory Still an Up-To-Date Measurement Instrument? Results of a Large-Scale Interview Study Application of Media Synchronicity Theory to Creative Tasks in Virtual Teams Using the Example of Design Thinking TrustyTweet: An Indicator-based Browser-Plugin to Assist Users in Dealing with Fake News on Twitter Application of Process Mining Techniques to Support Maintenance-Related Objectives How Voice Can Change Customer Satisfaction: A Comparative Analysis between E-Commerce and Voice Commerce Business Process Compliance and Blockchain: How Does the Ethereum Blockchain Address Challenges of Business Process Compliance? Improving Business Model Configuration through a Question-based Approach The Influence of Situational Factors and Gamification on Intrinsic Motivation and Learning Evaluation von ITSM-Tools fĂŒr Integration und Management von Cloud-Diensten am Beispiel von ServiceNow How Software Promotes the Integration of Sustainability in Business Process Management Criteria Catalog for Industrial IoT Platforms from the Perspective of the Machine Tool Industry Special Track 3: Demos & Prototyping Privacy-friendly User Location Tracking with Smart Devices: The BeaT Prototype Application-oriented robotics in nursing homes Augmented Reality for Set-up Processe Mixed Reality for supporting Remote-Meetings Gamification zur Motivationssteigerung von Werkern bei der Betriebsdatenerfassung Automatically Extracting and Analyzing Customer Needs from Twitter: A “Needmining” Prototype GaNEsHA: Opportunities for Sustainable Transportation in Smart Cities TUCANA: A platform for using local processing power of edge devices for building data-driven services Demonstrator zur Beschreibung und Visualisierung einer kritischen Infrastruktur Entwicklung einer alltagsnahen persuasiven App zur Bewegungsmotivation fĂŒr Ă€ltere Nutzerinnen und Nutzer A browser-based modeling tool for studying the learning of conceptual modeling based on a multi-modal data collection approach Exergames & Dementia: An interactive System for People with Dementia and their Care-Network Workshops Workshop Ethics and Morality in Business Informatics (Workshop Ethik und Moral in der Wirtschaftsinformatik – EMoWI’19) Model-Based Compliance in Information Systems - Foundations, Case Description and Data Set of the MobIS-Challenge for Students and Doctoral Candidates Report of the Workshop on Concepts and Methods of Identifying Digital Potentials in Information Management Control of Systemic Risks in Global Networks - A Grand Challenge to Information Systems Research Die Mitarbeiter von morgen - Kompetenzen kĂŒnftiger Mitarbeiter im Bereich Business Analytics Digitaler Konsum: Herausforderungen und Chancen der Verbraucherinformati

    We see we disagree: Insights from Designing a Cooperative Requirements Prioritization System

    Get PDF
    Information systems development is driven by a variety of stakeholders – each with specific requirements. Modern agile development methods, like Scrum, allocate the vital step of prioritizing requirements to dedicated roles like the product owner. However, this can create a bottleneck and may lead to misunderstandings and conflicts between stakeholders. Enabling stakeholders to cooperatively prioritize requirements frees up product owners while involving stakeholders more closely in a crucial development step, strengthening their ties to the final system. Therefore, stakeholders must form a mutual and shared understanding of requirements. This research in progress utilizes the design science research methodology to propose design principles for a cooperative requirements prioritization system using the MuSCoW method – which classifies requirements in four categories. By transferring the theory of shared understanding to the field of requirements prioritization, we derive design principles for cooperative requirements prioritization systems. We enable a group of heterogeneous stakeholders to identify and discuss differences and come to a mutually-agreed prioritization decision. We introduce design features that instantiate these principles and present promising results of an initial pre-test. This research in progress contribute to information systems development by introducing principles for the under-researched, but important class of requirements prioritization systems. Keywords: Requirements Prioritization, Cooperative System, MuSCoW, Desig

    Towards the Design of an Interactive Machine Learning System for Qualitative Coding

    No full text
    Coding is an important process in qualitative research. However, qualitative coding is highly time-consuming even for small datasets. To accelerate this process, qualitative coding systems increasingly utilize machine learning (ML) to automatically recommend codes. Existing literature on ML-assisted coding reveals two major issues: (1) ML model training is not well integrated into the qualitative coding process, and (2) code recommendations need to be presented in a trustworthy way. We believe that the recently introduced concept of interactive machine learning (IML) is able to address these issues. We present an ongoing design science research project to design an IML system for qualitative coding. First, we discover several issues that hinder the success of current ML- based coding systems. Drawing on results from multiple fields, we derive meta- requirements, propose design principles and an initial prototype. Thereby, we contribute with design knowledge for the intelligent augmentation of qualitative coding systems to increase coding productivity

    Accelerating Deductive Coding of Qualitative Data: An Experimental Study on the Applicability of Crowdsourcing

    No full text
    While qualitative research can produce a rich understanding of peoples’ mind, it requires an essential and strenuous data annotation process known as coding. Coding can be repetitive and time-consuming, particularly for large datasets. Crowdsourcing provides flexible access to workers all around the world, however, researchers remain doubtful about its applicability for coding. In this study, we present an interactive coding system to support crowdsourced deductive coding of semi-structured qualitative data. Through an empirical evaluation on Amazon Mechanical Turk, we assess both the quality and the reliability of crowd-support for coding. Our results show that non-expert coders provide reliable results using our system. The crowd reached a substantial agreement of up to 91% with the coding provided by experts. Our results indicate that crowdsourced coding is an applicable strategy for accelerating a strenuous task. Additionally, we present implications of crowdsourcing to reduce biases in the interpretation of qualitative data
    corecore